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A DISCUSSION OF MEASUREMENT ISSUES AND THEIR 
USEFULNESS AS INDICATORS OF STUDENT SUCCESS 



Key Conclusions And Recommendations ; 

• The Graduation Rate Is An Invalid Indicator Of Labour Market Success Of 
College Graduates 

• Do Not Use Graduation Rates For Provincial Funding 

• Start With Pilot Studies For Calculating the Graduation Rate Based On College 
Student Identifier Number 

• Launch A Survey Of Early College Leavers — Identify Successful From 
Unsuccessful Leavers 

• Develop A Province Wide Student Number System, Initiate A Systematic Long- 
Term Survey Of Students And Young Workers 
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1.0 EXECUTIVE SUMMARY 



Introduction and Terms of Reference 

Arthur Dormer and Fred Lazar were retained by the Association of Colleges of Applied Arts & 
Technology of Ontario (ACAATO) to identify, if possible, a valid and policy useful formula that 
links together for the Colleges the graduation rates and labour market success of their graduates. 

The assignment has a number of complex sub-components. We were asked to assess the 
usefulness and the validity of the “cohort-based” measure of graduation used by Ontario Colleges, 
and whether the current measures could be improved. We were also asked to consider whether 
the graduation rates of Colleges actually reflect the labour market success of graduates. Finally, 
we were asked to address a very important public policy question: whether it was appropriate to 
use the college graduation rate for Government fiscal allocation purposes. 

Fortunately, our analysis was able to tap into the considerable expertise that exists in the Colleges 
and the Ministry with respect to the KPIs, graduation rate calculations, and funding issues. We 
held two roundtable consultation sessions with college officials, one with the College Presidents, 
the other with officials directly involved in providing KPI information for measurement purposes. 

Ministry officials were very helpful to us all along the way both in terms of sharing data that they 
had compiled and in terms of setting out the Ministry position on KPIs and the graduation rate. 
Ministry officials also attended the two roundtable sessions. Finally, we undertook some special 
pilot studies and consultations with five Colleges with respect to their reporting on KPIs and the 
measurement of graduation rates at the program level. The pilot studies suggest that adapting a 
student tracking measure of graduation at the college level should not be too onerous. 

KPIs and the Graduation Rate 

The need for this research stems from an important new policy direction introduced by the 
Ministry of Training, Colleges and Universities (MCTU) that requires the Colleges to publish key 
performance indicators (KPIs) for the general public as well as for the Ministry. Other provinces 
in Canada and several state governments in the US have also introduced their own versions of 
KPIs in their jurisdictions. 

The four key performance indicators that were initially developed were the graduate employment 
rate for each College, the graduate satisfaction rate, the student satisfaction rate and the employer 
satisfaction rate. Three of the KPIs (the graduate employment rate; graduate satisfaction rate; and 
employer satisfaction rate) are presently tied in a small way to Provincial funding of colleges. A 
fifth KPI - the graduation rate and its alter ego, the retention rate - is also calculated by the 
Colleges. But a number of problems have arisen both with the calculation and the methods as well 
as with the interpretation and use of the data. Consequently, the college graduation rate is 
currently not tied to Ministry funding, though our consultations with college officials, 
stakeholders and other experts suggest that there is some risk that in the future the Ministry 
would also use the graduation rate for fiscal allocation purposes. 
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Clearly it is important that the KPIs, and in particular the graduation rate, accurately represent 
what is really going on and accurately reflect the interests of the key stakeholders - students, 
employers, the Government (MCTU specifically) and the 25 Colleges. It is also important that the 
graduation figures not mislead the college officials, the public, students, or the Ministry officials. 

Unfortunately, the current “cohort class” method for measuring college graduation rates is flawed, 
and the data are very misleading, particularly at the program level. Under the present system there 
is no tracking of individual college students, as is the case in the university measurements. As 
well, the Colleges use a shorter time frame than do the Universities in calculating graduation rates 
and the cohort methodology is backward looking rather than the more traditional forward 
looking. 

The tracking of individual students is clearly the direction to go in terms of monitoring college 
graduation rates. This is confirmed not only by the Ontario Universities’ experience, but also by 
the experiences of other jurisdictions. However, while the student tracking approach would 
provide superior information compared to the present cohort method, nonetheless in the college 
case, even this method could result in findings that are problematic and misleading, particularly 
with respect to the labour market achievements of graduates and early leavers from Colleges. 

College Graduation is an Invalid Indicator of Labour Market Success 

College graduation rates at the program level only partially correlate with short-term (i.e. 
immediate) labour market success, since we have no data relating to the employment experiences 
of the early leavers. But there is a fairly extensive literature explaining the apparent high level of 
college attrition (approximately 45% in Ontario), though the literature is limited in terms of 
dealing with the labour market success of college graduates compared to non-graduates from 
Colleges. 

The amount of information on successful versus unsuccessful leavers is very limited. However, 
there are many anecdotal examples of early leavers who are successful in the job market, 
particularly when the job market is hot. In other words, one cannot assume that attrition 
correlates with an unsuccessful job situation. The reverse might be the case, that the college 
experience, without formal completion of programs, was still a valuable input for the job market. 

Labour market success at the college level involves much more than formal graduation. In some 
parts of the job market, formal credentials, graduation and a certificate are necessary to qualify for 
a job. For other jobs, formal credentials are not required. Only some degree of training and work 
experience are necessary. 

Some Background Conclusions Derived from our Consultations and Research 

KPIs, Graduation Rates and Labour Market Success 
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Our research suggests that, with the exception of the graduation rate, that the four other KPIs 
provide some value added to the key stakeholders - the Colleges, MCTU, students, employers 
and the wider community. However, even these KPIs have only an immediate, not longer run, 
linkage to the labour market success of college graduates. 

For example, the graduation employment rate and the employer satisfaction rate are not 
monitored over longer time periods. Nor for that matter is the quality of the jobs (as reflected in 
incomes earned) a consideration in these measures; nor is upward mobility reflected in the 
Ministry survey data. Finally, the labour market experiences of early leavers (non-graduates) are 
also not taken into consideration with these measures. 

The KPI statistics gathered by the Colleges with the assistance of the Ministry also indicate that 
there is a link between student OSAP defaults and the graduation rate from programs. 
Consequently, there is a need to improve the graduation rate data, even if they have only limited 
use in evaluating labour market success. 

As already noted, the cohort approach for measuring graduation rates at the college program level 
is flawed and misleading. Nonetheless the cohort methodology currently used by the Colleges may 
be useful with respect to calculating aggregate graduation rates. But the figures are not useful on 
a program-by-program basis. Indeed, the figures at a program level jump all over the place (in 
some cases with graduation rates above 100%), suggesting that there are some major problems 
with the current measurement basis. 

Interpreting Comparative Graduation Rate Data 

The Colleges in Ontario are very diversified in terms of their program offerings. Since separate 
programs have very different graduation rates, the comparisons of aggregate graduation rates 
among Colleges are problematic. 

Nonetheless, with all of its flaws, Ontario’s aggregate college graduation rate (55%) compares 
favourably with the graduation rates in other jurisdictions. For example, the national US college 
graduation rate was 38% for two-year programs. The rate in Newfoundland was 50% and the 
graduation rates in Kentucky and South Carolina were 11% and 14% respectively. 

Similarly, comparing college graduation rates on a system wide basis with university graduation 
rates is also analytically misleading, though Universities reported a higher average graduation rate 
-71% compared to 55% for the Colleges. . 

The common problem in comparing graduation rates across different jurisdictions and between 
Colleges and Universities in Ontario is that different approaches and definitions are used. 

Why are College Graduation Rates Typically Lower than Universities? 

The literature, our consultations and the pilot studies all point to the same general explanations 
for higher average attrition at the Colleges compared with the Universities. Demographic and 
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socio-economic factors, combined with an open access policy at Ontario’s Colleges, are the key 
factors that explain the higher measured attrition rates and relatively lower graduation rates at the 
Colleges. In a nutshell. Colleges are more unique than Universities. 

• College students tend to be older than their university counterparts, and have different goals 
for entering Colleges. Students entering the Colleges from high schools or from the work 
force are more immediately required to make career choices than their counterparts entering 
Universities. 

• Some college students receive work experience as part of their college education, and are 
tempted by job offers while enrolled in their programs. 

• To a greater degree than is possible at the college level, students entering University can select 
a number of courses in their first year before they decide to specialize. 

• There is considerable diversity among the Colleges in terms of the student population, the 
program/course offerings, and the strength of the local labour market. 

• Colleges also stress easy entry access to programs relative to their university counterparts. 
One well-known researcher indicated that “Institutions that admit large numbers of less-well 
prepared students will tend to have low retention rates, regardless of how effective then- 
retention programs are”. Consequently, it is unfair to compare the retention rates of different 
types of institutions. 

• Finally, success at the college level involves more than graduating students. Indeed, in some 
programs, graduation and a certificate are necessary to qualify for a job. For other jobs, 
formal credentials are not required. Only some degree of training and work experience is 
necessary. 

Improving the Measurement of Graduation Rates at the Colleges 

Under the cohort measurement system, retention is described as “students who complete 
diploma/certificate programs within a minimum time frame expressed as a percentage of entering 
students.” Unfortunately, this graduation rate measure often provides misleading (and even silly) 
findings because of so much program switching, dropping in and out of the system and movement 
between Colleges, the labour market and Universities. 

The university tracking system assigns graduating students to the programs in which they first 
enrolled and thus is preferable to the cohort tracking system, which assigns graduating students to 
the programs from which they graduate. However, what do the resulting university program 
graduation rates tell us about the “success” of each program? Even though the university tracking 
system captures transfers between programs within the same institution, the reported graduation 
rates do not necessarily correlate with the ability of a program to enhance its students’ labour 
market opportunities. 
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Another problem arises when a program requires as a pre-requisite for entry a year or more of 
general education or another program. In such cases, it is important to properly define the entry 
year. If the entry year is defined as the year in which a student starts the program rather than the 
year a student enters into a College or University, then graduation rates will tend to be higher, 
ceteris paribus, than for programs where the entry year is the one in which a student first enrolled 
in the post-secondary institution. Aggregate institution graduate rates also will be higher where 
the entry point is defined as the year in which a student enters a program. 

The introduction of a university style measure of graduation rates at the program level (i.e. 
following individual students rather than cohorts) would be an improvement over the current 
system. But recognize that using a college institution identifier will still face the problem of 
capturing and measuring students moving between different Colleges and in and out of the college 
system (either to the job market or to other post-secondary educational opportunities). 

In closing, while a university style student tracking system is superior to the current cohort 
approach used by the Colleges, its findings might also prove problematic. 

Our Key Conclusions und Recommendations 

General Conclusions And Recommendations 

• The graduation rate is an invalid indicator of the labour market success of college graduates. 

• Do not use graduation rates for provincial funding at this time. 

• Start with pilot studies for calculating the graduation rate based on college student identifier 
numbers. 

• Launch a survey of early college leavers to identify successful Irom unsuccessful leavers. 

The research and evidence we have marshalled suggest that the college graduation rate is an 
invalid measure of labour market success. Indeed, the measured college graduation rates have 
little bearing on the labour market success of graduates and/or the early leavers. Even if the 
graduation rate is better constmcted, we would still worry that the information would be of 
limited use to outsiders and could be easily misinterpreted. 

Consequently, the graduation rate should not be used for Ministry funding. In effect, we have to 
look at measures other than the college graduation rate to better reflect labour market success. 

It is still worthwhile, however, to improve the measurement of graduation rates at the College 
level for internal college planning purposes. Consequently, ACAATO should propose that MTCU 
move towards introducing a student tracking measurement system for graduation based on 
college student identification numbers. 
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To ensure that the graduation rate data are reliable, ACAATO should suggest that MTCU 
sponsor several pilot studies, experimenting with starting and ending dates (i.e. the starting point- 
semester one or two, and the termination point, up to five years). 

ACAATO should also petition MTCU to undertake a survey of early leavers from college 
programs, attempting to identify successful from unsuccessful college leavers in terms of the job 
market prospects. We believe that this kind of a survey would provide value both to the Colleges 
and to the Ministry. 

A Better Method for Linking Retention to Labour Market Success 

Our research suggests that the Ontario labour market is very dynamic, and that new entrants into 
the labour market quite properly experiment (job hop) before settling down into a career/job path. 
There is a counterpart to the labour market/job hopping when one considers student mobility 
between different college programs and within and between Colleges and Universities. All of this 
complicates the measurements and interpretation of graduation statistics. 

Accordingly, the optimum approach would be to establish a provincial wide student identification 
number, and use it to follow the education experiences and job paths of Ontarians. The provincial 
wide number approach also allows for a much longer time period in terms of tracking the labour 
market experience of those who have graduated and of those who are truly early leavers from the 
Colleges. This long-run tracking approach may solve the problem of the inadequacy of the 
graduation rates. It is only when we have a better grasp on the job experiences of early leavers 
and graduates that we can then assess the labour market validity of college programs. 
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2.0 



INTRODUCTION 



2.1 Objectives 

Accountability in post secondary education and training institutions and programs has become 
more prominent for funding decisions by the Ontario Government, as well as in many other 
jurisdictions in recent years. In the US, the combination of increased costs along with decreased 
state resources have resulted in public demands that college and university administrators take 
steps to ensure greater quality, productivity and effectiveness of their institutions. Colleges and 
Universities are being asked to be more responsive to state concerns and to be publicly 
accountable to stakeholders - students, parents, employers and the general public. Several states 
have adopted performance measures to respond to these accountability demands. 

Historically, Ontario’s Colleges have had a tradition of being accountable through methods such 
as community-based governance, program advisory committees and provincial program standards. 
In their search for quality. Colleges build into their internal processes opportunities to review and 
reflect on the nature, content and delivery of their programs. Perceptions are sought from 
students, graduates, employers of graduates, advisory committees, faculty and staff of the College 
and members of the college’s Board of Governors. These inputs are synthesized with statistical 
reporting processes to create a view of the college’s effectiveness in carrying out its mandate. 

Lacking, however, has been a system-wide accountability framework that actually measured 
colleges’ performances against goals identified by the ministry responsible for postsecondary 
education and the Colleges. Accountability requires a set of objectives for these institutions and a 
means for measuring the success for each institution in achieving the defined public policy goals. 
In Ontario, the provincial government has indicated that it intends to depend more heavily on Key 
Performance Indicators (KPIs) in its future policy and funding decisions. Five KPIs have been 
identified thus far: post-college outcomes, graduate satisfaction, employer satisfaction, student 
satisfaction and graduation rates. 

It is important that the KPIs provide the information needed to assess whether the institutions are 
achieving the goals set by the government. In the case of the 25 Colleges, which comprise 
ACAATO, the KPIs should reflect what is important for students, employers and the taxpayer and 
accurately represent the degree of success that each College has in serving the needs of each 
stakeholder group. 

Accountability should be centered in the mission of the college system and should respect the 
diversity of each College in Ontario. Internal accountability processes can be strengthened by a 
well-structured and complementary system of external accountability measurements - the right set 
of KPIs. But these KPIs need to be clear and explicit and relate performance to purpose and 
expectations. Measuring performance is an ongoing, evolutionary process that should contribute 
to the improvement of the quality of what Colleges do. 
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In 1998, the Colleges worked collaboratively with the MTCU to implement the first public sector 
key performance indicator project in Ontario. The following are the four key performance 
indicators that were developed through the joint effort of the Colleges and the MTCU: 

• Graduate employment rate of each College; 

• Graduate satisfaction rate of each College; 

• Student satisfaction rate with the Colleges; and 

• Employer satisfaction rate (i.e. satisfaction with the generic/vocational skills of their 
employees who graduated from the Colleges). 

Data collection on graduate outcomes, graduate satisfaction and employer satisfaction began in 
the Fall of 1998. The results released in 1999 demonstrated that the Colleges in Ontario provide 
the quality education and training that Ontario needs in the knowledge economy. 

These three KPIs (graduate employment rates, graduate satisfaction and employer satisfaction) 
will be factored into the mechanism for distributing government transfer payments to the Colleges 
starting in fiscal year 2000-01. Commencing with this fiscal year, 2% of the allocation of funding 
for operating grants for the Colleges will be based on these KPIs. The comparisons will be across 
the 25 Colleges in Ontario. No comparison will be made between the Colleges and the 
Universities, at least at the outset. Student satisfaction and student graduation data, also being 
gathered beginning with the 1998-99 academic year, will not be tied to funding distribution at this 
time. 

It is important to keep in mind that when the link between KPIs and funding begins, 98% of the 
funding will continue to be based on total college enrolment. If a College attracts students, it also 
will attract money from the province. Eventually these three indicators will account for up to 6% 
of total funding for the Colleges. 

Funding models are now being evaluated to ensure that the KPIs serve the ultimate objectives of 
program and service improvements at the Colleges. For example, the government would like to 
see a 100% employment rate for college graduates and will reward the Colleges that succeed in 
placing their students in jobs. But should Colleges located in regions with unemployment rates 
above the provincial average be penalized if they attain marginally lower graduate employment 
rates? 

Several questions have been raised about the graduation rate KPI. For example, it is difficult to 
define and measure and the interpretation of this variable in respect to the roles of the Colleges 
and the public policy goals set out for these institutions is ambiguous. 

Since there are considerable and legitimate concerns over the measurement and interpretation of 
the graduation rate KPI measure, the provincial government is not yet using it for fiscal allocation 
purposes. Moreover, the Colleges are not required to publish anything other than the aggregate 
college graduation rate with the other KPI information that is being made available. 
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Nevertheless, the fact that the 25 Colleges in Ontario are not homogeneous either in programs, 
geographic locations, their history and their student demographics complicates the interpretation 
of broad brush KPI measures. Thus, to be useful, the KPIs, especially the graduation rates should 
allow for differences among the Colleges and between the Colleges and other post-secondary 
institutions, in particular the Universities. 

Hence, the purpose of this research project is to examine the feasibility of using the graduation 
rate for funding allocation decisions, and if so, to identify for the Colleges and the Ministry a 
policy useful formula that links together graduation rates and the labour market success of 
graduates. 

As a starting point, one can ask whether it is appropriate to equate the graduation rate 
with labour market success? Statistics Canada data suggest that this is indeed the case. 
Labour market survey data reveal that there is an inverse correlation between education 
and unemployment rates and a positive correlation between educational attainment and 
incomes. 

But does this mean that it is right to assume that non-graduation equates to labour market failure? 

The traditional concept of the graduation rate, regardless of how it is measured, does not 
necessarily provide a reliable indicator of the “success” of the Colleges in achieving the 
government set goals. The number of graduates and the program and aggregate graduation rates 
do not reflect those students who have had a positive experience in College and who may have 
secured employment related to their area of education and elected not to finish their programs; 
temporarily gone back to work only to continue with their studies at a later time on either a full- 
time or part-time basis; or, decided to continue their programs on a part-time basis. Thus, failure 
by a student to complete a college program within a fixed period of time does not necessarily 
mean that such students have not benefited from their education or that they have not improved 
their labour market opportunities and status. 

Experimentation with jobs and college programs and continual movement between the labour 
market and the education market characterise a dynamic economy where technological change 
requires a flexible, mobile and adaptable work force. The possibility for a life-long education 
experience plays a key role in ensuring that the work force has these important characteristics. 
This, in turn, has important implications for defining and interpreting graduation rates and for an 
education system where students complete the requirements for their degrees/certificates/ 
diplomas and at the same time complete their formal education. 

2.2 Role of the Colleges 

As a starting point in any discussion of the purpose and value of KPIs in general, and the 
graduation rate in particular, it is useful to identify the role of the Colleges in the post-secondary 
education and training system in Ontario. While identifying the role of the Colleges will depend 
upon which group of stakeholders - students, employers, administrators, tax payers, government 
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- one asks, there is likely to be agreement that a key role, if not the primary role, is to develop 
and augment the stock of human capital in the province. 

The specific role identified for the Colleges of Applied Arts and Technology of Ontario is to 
develop graduates who have achieved learning outcomes that are consistent with broad-based 
employment and societal needs and who are able to adapt to changing employment/workplace 
demands, within an environment that facilitates student learning and promotes personal and 
professional growth. 

Since a fundamental role of Colleges is to train people for a continually changing labour market, 
then graduation from a program may not necessarily correlate with the degree of success that 
Colleges have in fulfilling this role. Developing human capital may not require the traditional 
linear model of education, where a student enters, completes the program and then permanently 
leaves. 

Colleges will be successful in fulfilling their role if they allow their students to integrate life-long 
learning and training with job experience. Flexibility requires students/workers to be able to have 
or attain the skills and training needed to fill a large number of jobs during their careers. It will 
become increasingly rarer for an individual to enter the labour market after graduation from some 
education program and to work for the same employer and in a similar job role throughout her/his 
work-life. 

Colleges will be facilitating labour market flexibility and will be supporting a dynamic economy by 
providing an environment where individuals can drop in whenever they need skills upgrading or 
retraining, and drop out when they have achieved their personal labour market goals, only to re- 
enter again at a later date when a future need arises. In this world, and it is the one that is 
evolving, and for countries to be successful they must accommodate their social and 
education infrastructure and policies to it, the linear model leading to graduation within a 
set time period may become increasingly outdated. 

As a result, the traditional concept and measurement of the graduation rate may be compatible 
with the education/training model that was or currently is, but not with the model that should and 
will be. Current trends in the nature of work/jobs and the pace of workplace change suggest that 
students will increasingly demand only the appropriate “bite-sized” educational modules that fit 
their specific needs. Increasing demand for modularized education suggests that to measure 
performance in completion of programs, many with rigid start and finish points, might be focusing 
on a diniinishing model. An alternative model would measure course completions against course 
attempts. 

2.3 Defining Success 

There are a number of stakeholders in the college system. As a result, success is a multi-faceted 
concept. Defining success depends upon what the role is of the college system as a whole, and of 
each College within the system. And the definition will depend upon which of the stakeholder 
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groups one asks. So, it is unreasonable to expect that one indicator can provide a good measure 
of how successful Colleges are in achieving their goals. 

Thus, that there is controversy over the definition, measurement and value of graduation rates is 
to be expected. 

As one commentator has asked: is success to be measured by the ability of a post-secondary 
institution “to keep clients captured for the full period of their sentence?” Furthermore, 
how do you measure success for programs where graduation is a failure; for example, fine 
arts where if students have not been picked up for a role, then they have not succeeded? 

Students get work experience as part of their college education, and so they may be 
tempted by job offers while enrolled in their programs. Employers have experience with 
students and so may be more likely to offer them jobs. In the following areas - printing, 
graphic design and animation - employers are not waiting for students to graduate before 
offering them attractive jobs. 

Therefore, stopping out may not necessarily have negative repercussions for students and society 
as whole. Students can drop out to take advantage of job opportunities and may return at a later 
date to complete their programs or transfer into new programs. Furthermore, students in Colleges 
are more likely to experiment with courses and programs than students in Universities because, 
with the exception of the professional programs at the Universities, Colleges are more career- 
oriented. 

One of the leading researchers in the field of student attrition, Vincent Tinto, suggests that the 
term “drop out” be eliminated from any future discussions about student withdrawals. His 
reasoning for not using the term is that many community college students can “stop out” from 
their studies (i.e. leave for a period of time and then at some point return), or they can transfer to 
other post-secondary institutions or training programs, or they can be required to leave because of 
misconduct, or the can just quit their studies altogether. The singular term “drop out”, as used in 
our post-secondary system, can hinder a full understanding of the various types of “early leavers.” 
(Working Group Report, p. 5) 

Each College has a different mix of students, faces different labour/economic market conditions 
and serves different social needs. Thus, Colleges take different paths to achieve their goals — and 
so success for each College is difficult to define. 

Nevertheless, graduation rates, if properly defined and measured, may serve a role. But alone and 
without clarification, these rates do not measure success for the Colleges. For example, Ontario 
Colleges had an average system- wide retention rate of 55% in the 1997-98 academic year. The 
graduation rate for the US for two-year Colleges was 38% in 1997-98. Does this mean that the 
Colleges in Ontario are more “successful” than the two-year Colleges in the US? 

As well, in 1997-98 the average graduation rate for Universities in Ontario was 71%. Do 
Universities better fulfill their mandates than the Colleges in Ontario? Which group better serves 
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the needs of society? Is it proper to equate the low graduation rate at some Colleges with labour 
market failure? Alternatively, is it right to assume that non-graduation equates to labour market 
failure? 

In other words, a major policy issue relates to defining what is a successful graduation from a 
College. Is it finishing/receiving a diploma or is it generating a job in a program-related field? 



College 


Graduation Rate 


Algonquin 


61% 


Boreal 


73 


Cambrian 


50 


La Cite 


46 


Canadore 


61 


Centennial 


49 


Conestoga 


63 


Confederation 


72 


Durham 


55 


Fanshawe 


57 


Georgian 


61 


George Brown 


59 


Grand Lacs 


67 


Humber 


53 


Lambton 


45 


Loyalist 


61 


Mohawk 


49 


Niagara 


53 


Northern 


43 


St. Clair 


37 


St. Lawrence 


54 


Sault 


51 


Seneca 


46 


Sheridan 


71 


SS Fleming 


62 


Provincial Average 


55 



There also was considerable variation in the average graduation rates across Colleges and 
Universities. Among the 25 Colleges, the lowest aggregate rate was at St Clair at 37% and at 
Northern at 43%, while the highest rate was at Boreal at 73%. Carleton had the lowest graduation 
rate (46%) among the Universities, with Queen’s having the highest (87%). By ranking the 
Colleges and Universities in descending order of graduation rates, can one conclude that the 
resulting rank order is appropriate in terms of their relative degrees of labour market success or 
training? We do not think so! 
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Colleges and Universities differ in many significant ways, including their pedagogical, social and 
economic roles. Accessible career programs or courses coupled with relatively lower tuition fees 
encourage students to enroll in Colleges who may be academically underprepared or ambivalent 
about a post-secondary education. Attrition is higher for students have not completed their OACs 
and the open access policy for Colleges results in many such students enrolling in Colleges in 
Ontario. As well, courses feared towards updating the skills of employees of local businesses may 
attract students who do not intend to complete the full career programs. 

Further, the socio-economic demographics of students - age, parental responsibility, income, 
education attainment of parents and secondary level education - differ between Colleges and 
Universities. The average age of Ontario college students is 26, more than 60% do not come 
directly from high school, over 55% require financial aid and double the number of single, sole- 
support parents attend College than University in the province. 



University 


Graduation Rate - 6 years 


Graduation Rate - 7 years 


Brock 


71% 


72% 


Carleton 


44 


46 


Guelph 


67 


69 


Lakehead 


59 


61 


Laurentian 


54 


56 


McMaster 


74 


75 


Ottawa 


64 


66 


Queen’s 


86 


87 


Toronto 


61 


63 


Trent 






Waterloo 


75 


77 


Western 


74 


74 


Wilfrid Laurier 


81 


82 


Windsor 


55 


56 


York 


58 


60 


Ryerson 


56 


57 


Nipissing 


54 


56 


OCA 







As a result, graduation rates should differ between Colleges and Universities. Therefore, 
comparisons of graduation rates across programs, across Colleges and between Colleges and 
Universities should be based on some form of indexing of the reported graduation rates to 
normalized or anticipated rates predicated on some correlation with one or more measures of 
success. 

The non-indexed, graduation rates could be useful for internal planning within Colleges. They also 
may be useful as one of many inputs into policy-making decisions by the provincial government. 
However, they may be entirely misleading to the public and lead to poor decisions by students 
considering their post-secondary education choices. 




17 



16 



Moreover, Colleges should not base their longer-term plans on measures of success which 
may be backward looking (in order to maintain their government funding), but rather on 
measures which more accurately reflect their longer term economic and social objectives. 

2.4 Outline of the Report 

The Report is organized as follows. Section 3 discusses a number of issues associated with 
defining and measuring graduation rates. The issue is discussed not only at an empirical level, but 
also in terms of assessing the policy distortions that could occur when comparing graduation rates 
within the college system and between Colleges and Universities. 

Section 4 discuses KPIs and expands on the discussion of the graduation rate measure. Lessons 
from other jurisdictions and some discussion of the relevant literature relating to measurement 
issues and the labour market experience of College and other post secondary graduates are also 
presented in section 4. 

Section 5 considers graduation measurement issues as they relate to five Colleges in Ontario - 
Canadore College, Humber College, Boreal College, Sir Sanford Fleming and Fanshawe College. 
The main point for the case studies is to indicate to ACAATO the level of progress being made on 
measuring college graduation at a student level. 

The final section of the Report sets out our broad conclusions with respect to measurement issues 
and policy concerns. The recommendations in this section are intended to provide ACAATO and 
the Colleges with a useable and practical direction for improving the measurement and the use of 
college graduation statistics. 

The appendix section, which is attached at the end of this Report, provides some of the basic 
background data and information discussed in our study. The data that are reviewed and 
discussed in the appendix includes KPI system data for the Colleges and the Universities, the 
measurement of graduation rates in other jurisdictions, the five case studies and the program 
graduation rate at three universities - Carleton, Queen’s and Guelph. 
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3.0 



RETENTION RATES 



3.1 Definition 

According to a MTCU/ACAATO joint statement, retention is described as “students who 
complete diploma/certificate programs within a minimum time frame expressed as a percentage of 
entering students.” The above definition is the crux of the measure currently being used. The 
cohort method is currently used to measure this definition of graduation for the Colleges. The 
graduation rate also can be measured by tracking individual students and aggregating over all the 
students that enter into a College/University and/or program in either post-secondary institution. 

Whichever methodology is used, the resulting graduation rate estimate depends upon when you 
take the picture; that is, the time period after commencement of the program. The longer the time 
frame, the higher and the more reliable will be the resulting graduation rates. 

Furthermore, the graduation rates will be more accurate when they are aggregated over programs 
rather than measured on an individual program basis since this will capture students who transfer 
between programs. 

3.2 Measuring Graduation Rates; Universities 

Universities assign their own identification numbers to each student who enrolls and track each 
student, using these numbers, for up to seven years after first enrolling. The university graduation 
rate is measured in the following way. It is the percentage of students who enroll in a bachelors or 
first professional degree program in year X who subsequently receive a university degree between 
year X+1 and X+7. The data are compiled by the Ministry and verified by the separate 
Universities. 

Graduation rates also have been calculated by relating the number of students who enroll in year 
X who subsequently receive a university degree between year X+1 and X+6. As expected, the 
graduation rates based on this measure using a shorter time frame are lower. They are lower by 
one to two percentage points according to the graduation rate data for the universities (see table 
above). 

Each University has its own student identification number. There are no province-wide numbers. 
Therefore, the tracking system is unable to follow students across Universities, and so the 
resulting graduation rates may be underestimated for some Universities and over-estimated for 
others. For specialized programs at the undergraduate level, the starting point for tracking 
students for the purpose of measuring the graduation rates is a student’s second year of study. 
Year one is considered a transition year in which students take general education courses before 
they decide on their area of specialization. Since a significant number of students drop out of 
University during their first year, graduation rates for specialized undergraduate programs will 
tend to be higher than if the starting point included all students who entered in first year. 
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In measuring the graduation rates by program, if a student starts off in Physical Education for 
example, but later transfers into General Arts and graduates from this program, the student is 
counted as a graduate of the Physical Education program. 

3.3 Measuring Graduation Rates: Colleges 

Colleges use the reverse cohort method to measure graduation rates; that is, they take the ratio of 
the total number that graduate in year X to the total number that enrolled in a one-year program 
in year X-1 plus those who enrolled in a two-year program in year X-2 and those who enrolled in 
a three-year program in year X-3. In other words, there is a single graduation year and multiple 
intake years. So for starters, the Colleges use a shorter time frame on average than do the 
Universities. They do so because college programs are shorter than those in Universities. 

The cohort method provides a reasonable measure of the graduation rate at the college-wide 
level, but not at the program level. But even the aggregate college graduation rate is deficient 
because it does not capture students transferring between Colleges or between a College and a 
University or between a College and the labour market and back again to a College. 

Moreover, the program mix is an important driver of graduation rates. One-year programs are 
likely to have higher graduation rates than three-year program, and this difference is likely to be 
compounded by the cohort method used by the Colleges to measure graduation rates. 
Consequently, if Colleges were to shorten program length, they could conceivably increase 
graduation rates. But would this change necessarily reflect greater success by the Colleges in 
achieving their goals? 

While the plan of the MTCU is to make individual programs at Colleges somewhat accountable in 
terms of student success; nonetheless, comparisons of the graduation rates across programs are 
not reliable. Colleges offer a much larger number of programs than Universities. For example, 
Carleton (with a graduation rate of 46%) listed 14 starting programs and Queen’s, with a 
graduation rate of 87%, listed 16 programs. By comparison, Humber College, with an aggregate 
graduation rate of 53% listed 83 programs and Boreal with a graduation rate of 73% listed 53 
programs. Thus, there is a greater opportunity and indeed and a higher propensity for students to 
switch between programs in Colleges than in Universities. 

Furthermore, the inclusion of the General Arts and Science (GAS) program in the graduation rate 
calculations for the Colleges is problematic since Colleges use this program as remedial area for 
students. The intent is not for students to graduate from GAS but to apply to core programs of 
choice when they are qualified through the GAS studies. 

Conestoga has an aggregate graduation rate of 63% when the GAS students are included in the 
calculation. Excluding these students increases the graduation rate to 68%. Sheridan College has a 
graduation rate of 78% when the GAS students are excluded, compared to the reported 71% 
graduation rate. The graduation rate for GAS at Sheridan is only 13%. The graduation rate for 
GAS students at Lambton is 27%. This compares to the aggregate graduation rate of 45%. 
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As well, in Universities it does not matter in what program a student graduates, but rather the 
program where the student starts for purposes of measuring graduation rates by programs. For 
Colleges it matters in what program the student graduates. Hence, we can find many examples 
where some college programs report very low graduation rates, while others have graduation 
rates of 100% or higher. 

3.4 Problems in Measuring Graduation Rates 

Graduation rates by program are problematic. The university tracking system assigns graduating 
students to the programs in which they first enrolled and thus is preferable to the cohort tracking 
system, which assigns graduating students to the programs Ifom which they graduate. However, 
what do the resulting university program graduation rates tell us about the “success” of each 
program? Even though the university tracking system captures transfers between programs within 
the same institution, the reported graduation rates do not necessarily correlate with the ability of a 
program to enhance its students’ labour market opportunities. 

Another problem arises when a program requires as a pre-requisite for entry a year or more of 
general education or another program. The GAS program at Colleges is a case in point. In 
addition, the Colleges have increased the number of programs which provide future options for 
students. In such cases, it is important to properly define the entry year, especially since 50% of 
students who leave Colleges during the first year do so in the first semester. 

If the entry year is defined as the year in which a student starts the program rather than the year a 
student enters into a College or University, then graduation rates will tend to be higher, ceteris 
paribus, than for programs where the entry year is the one in which a student first enrolled in the 
post-secondary institution. Aggregate institution graduate rates also will be higher where the entry 
point is defined as the year in which a student enters a program. 

Many of the anomalies that occur at the program level, especially with the cohort system, should 
be aggregated out at the institution level. However, both systems fail to capture transfers between 
institutions both within the college system and the university system and between the two systems. 
The university tracking system underestimates the graduation rates for each University. On the 
other hand, the college cohort system can underestimate aggregate graduation rates for some 
Colleges and overestimate for others. 

Another issue that arises is the appropriate time Ifame to use for measuring graduation rates. The 
longer the time period, the higher should be the resulting graduation rates. But does a longer time 
frame necessarily correlate with “success” for both students and post-secondary institutions? 

Students enrolling in University are less likely than students enrolling in Colleges to move 
between school and full-time work. Thus, university students should take less time to complete 
their formal programs and receive degrees than college students in their programs. As well, 
university students are more likely to be committed to completing their programs (i.e. to receive 
the degree) than are college students. University students may need a degree in order to improve 
their labour market possibilities. College students, on the other hand, may only need some 




21 



20 



technical courses to enhance their labour market prospects and may need to periodically upgrade 
their skills. Formal accreditation is less an issue in the job market for College students. 

Therefore, is a longer time frame more appropriate for Universities than for Colleges? Perhaps 
each student should have a different time frame based on his/her expectations at the time of entry 
into a post-secondary institution. 

A common tracking period for all programs leads to longer possible periods for graduation for 
shorter programs. In addition, many entrants into post-secondary institutions have clear goals 
when they enter. Others are unclear and are likely to experiment with both programs and work. 
Students entering from high schools are generally required to make career choices when selecting 
their college programs, whereas in Universities they can select a number of courses in their first 
year before they decide to specialize. College students may be more likely to change their minds 
regarding their programs of study or even whether completing a college program is necessary. 
Consequently, the same starting point may be inappropriate for all students, and the same end 
point (“X” years after the starting point) may also be inappropriate. 

3.5 Other Problems in Comparing Graduation Rates Between Colleges and 
Universities 



There are significant differences between Universities and Colleges in terms of their roles, their 
objectives and the socio-economic demographics of their students. And so, one should expect 
differences in graduation rates and these differences should not necessarily reflect differences in 
success. 

Colleges are more unique than Universities. The graduation rate measure is an output measure, 
and downplays the input side of the measure. Success at the College level is more than simply the 
graduation rate. 

Open and easy access to Colleges makes a big difference on the input side. The student body at 
Colleges consists of older students, more students who are married and more single parents. 
These types of students are more likely to pursue a post-secondary education to upgrade their 
skills and to accept a full-time job offer as soon as they have achieved a given amount of 
upgrading rather than wait until they graduate from their programs. 

The open access mandate for Colleges results in their having a number of students who require a 
considerable amount of remedial education. Special needs students may reduce measured 
graduation rates even though the Colleges are serving their social goals by helping these students 
prepare for the job market. 

Unlike Universities, Colleges do not need to sell a product that is completed. In some programs, 
graduation and a certificate are necessary to qualify for a program-related job. In others, some 
skills training without formal completion is acceptable to employers. 
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One of the key roles of Colleges is to train students and help them upgrade their skills. Therefore, 
it is not surprising for college students to drop in and drop out. In periods of low unemployment 
and tight labour markets, college students are more likely to drop out and accept job offers related 
to their areas of study. As well, in some college programs success requires students to drop out 
and accept job offers. 

There is now a closer association between Colleges and Universities than in the past. That 
is, more students are using Colleges as pre-university preparation and move to Universities 
after one or two years of study at a College. These students do not show up in the 
graduation statistics of the Colleges, even though they move onto Universities and possibly 
graduate from these institutions. At a minimum. General Arts and Science programs at 
Colleges, which serve primarily a preparatory role, should be excluded when measuring 
aggregate college graduation rates. 
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4.0 



KPIs 



4.1 Lessons from other Jurisdictions 

The Ministry of Training, Colleges and Universities shared with us some data that were compiled 
on the measurement of KPIs in other Canadian and US jurisdictions. Summary tables relating to 
these findings are set out in the Appendix 2 section. 

Alberta has implemented an individual tracking based graduation rate measure. The program 
completion rate is measured as the percentage of the entry cohort who complete the program 
within the normal program duration plus one year. The measured completion rate includes those 
who complete a program and those who transfer to and complete other programs within the 
College. The number of non-completers still active is also reported. Individual tracking of 
students is done only at the institution level using student institutional identification numbers. 
System-wide tracking of students cannot be done yet, but may be possible in two years time. 

Alberta also has a common KPI system composed of 1 3 measures for all of its post-secondary 
educational institutions. As in Ontario, the KPIs are linked to funding allocations. Alberta does 
not follow system wide student tracking, but seems to use a cohort measure similar to Ontario’s. 

British Columbia has a system that tracks students based on a province wide personal education 
number and calculates course completion rates since Colleges in BC serve as feeders into the 
Universities in the province. The course completion rate is measured as the number of stable date 
registrations (usually three weeks into term) in courses who have grade of pass or better. BC has 
a high course completion rate of 80%. Note that BC excludes adult basic education, adult special 
education and English as a second language from the calculations. 

BC also has a system of 26 KPIs that are intended to measure a wide range of outcomes 
associated with the achievement of system goals and objectives. The KPIs have been selected on 
the basis of reliability and usefulness. The following principles guided the selection of the KPIs in 
BC: 

• Consistent data definitions and collection across college system; 

• Emphasis on measuring results (outcomes and outputs); 

• Valid measures of what they are intended to measure; 

• Reliable over time; and 

• Clear, understandable and transparent. 

The general categories for the KPIs are: 

• Relevance and Quality - ensure learners are able to acquire the skills and knowledge 
needed; ensure learners acquire the necessary competencies to function successfully in life 
and work; ensure learners are able to acquire job readiness and job specific skills; 

• Access - ensure opportunities exist for participation in post-secondary education and 
training and that opportunities in all regions reflect regional education and training needs; 
eliminate attitudinal and physical barriers affecting participation of non-traditional learners; 
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promote equity of access across the system; improve opportunities for successful 
completion of post-secondary education; and 

• Affordability - ensure post-secondary education is affordable to students; ensure post- 
secondary education is affordable to taxpayers. 

Newfoundland measures the graduation rate for the sole College in the province by taking the 
number of graduates in a given year as a percentage of the total number of entrants within normal 
program duration. The graduation rate calculations use student information numbers. Students 
who leave before the academic prejudice date are excluded from the calculations. The graduation 
rate is 50% in the province. 

Two US states, Kentucky and South Carolina, use social security numbers to track students and 
to calculate graduation rates. 

Both South Carolina and Kentucky calculate the graduation rates by taking the first-time, full-time 
degree-seeking cohort of students who graduate within 150% of program duration. South 
Carolina excludes from the calculation of the graduation rates students taking two or more 
remedial courses since completion of programs by these students may take more than 150% of 
program duration. Kentucky also calculates the persistence rate: the percentage of full-time, 
degree-seeking freshmen who either graduated, transferred to another public institution or were 
still enrolled at the original institution after 150% of program duration. The graduation rate in 
Kentucky isl 1%. The persistence rate, which is more comprehensive, is 46%. The graduation rate 
in South Carolina is 14%. 

The US National Center for Education Statistics (NCES) published a survey of all funded two- 
year college programs. NCES is the benchmark source for graduation rate calculations in the US 
and its benchmark is to calculate the graduation rate for a cohort within 150% of normal program 
duration. Thus, for a two-year college institution, graduation would be calculated based on three 
years after entry (i.e. the number of students who enter in year X and the number who graduate in 
year X+3). 

In closing, comparisons of graduation rates across jurisdictions are problematic because different 
approaches and definitions are used. However, Ontario’s aggregate college graduation rate (55%) 
compares favourably with the graduation rates in other jurisdictions. For example, the national US 
two-year college graduation rate was 38%. The rate in Newfoundland was 50% and the 
graduation rates in Kentucky and South Carolina were 11% and 14% respectively. 

4.2 Ontario 

The proposed accountability framework for Colleges in Ontario is intended to consist of a two- 
tier system of performance indicators: those which derive from the shared goal and common 
outcomes of the college system and those which reflect the mission, values, and strategic direction 
of the individual Colleges. 
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The first level is comprised of five performance indicators that will be measured at each College 
using standardized definitions and measurement tools. They include: 

• Student Satisfaction 

• Post-College Outcomes/Graduate Success 

• Graduate Satisfaction 

• Employer Satisfaction 

• Quality Assurance. 

The suggested performance indicators for the second, equally important, level are intended to 
provide Colleges with sufficient flexibility to relate the Accountability Framework to their 
individual mission/strategic direction: 

• Access 

• Fiscal Responsibility 

The objective for the Student Satisfaction indicator is to measure performance against the goal of 
quality education and the student success-related goal of meeting the needs of students. The 
objective is to capture a variety of information from the students’ perspectives, including their 
views regarding the quality of their learning experience; its relevance to future employment, 
further education and life beyond the workplace. 

The Post-College Outcomes/Graduate Success indicator is intended to measure performance 
against the student success-related goals of: finding employment and continuing with educational 
pursuits. The objective is to capture information in order to determine whether alumni are able to 
find appropriate employment within a reasonable time frame after leaving College and/or alumni 
are able to gain entrance to further post-secondary studies of their choice. 

The Graduate Satisfaction indicator is intended to measure performance against the student 
success-related goal of meeting the needs and objectives of students. The objective is to determine 
the opinion of college graduates regarding their college experience, including their views as to 
whether their College helped prepare them for a specific occupation, a specific field of work, 
work generally, further educational pursuits, better citizenry, or meeting other personal objectives. 

The Employer Satisfaction indicator is intended to measure performance against the goal of 
quality education and the student success-related goal of student employability. It is important 
that employers are satisfied with graduate preparedness, program relevance, and the Colleges’ 
ability to meet emerging employer needs. Measuring the rate of employer satisfaction will provide 
an indication of how well employer needs are being met by the college system. 

The Quality Assurance indicator is intended to demonstrate that the College has and uses internal 
accountability processes to ensure the quality of college programs. This graduation rate serves as 
the proxy for this indicator. 
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The purpose of institutional performance indicators would be to acknowledge the diversity and 
the particular strengths of each College in order to support the individual Colleges in moving 
towards their stated strategic directions. The two general institutional performance indicators 
would be Access and Fiscal responsibility. 

Access issues would include the ability of students to pursue their chosen program of study within 
a reasonable distance from their home, without undue regard for individual disabilities or other 
personal factors, within a time frame which is reasonable responsive to individual backgrounds, 
circumstances and needs. 

Fiscal responsibility refers to a College’s use and management of its financial resources such as its 
administrative efficiency and its ability to obtain, organize and administer resources so that 
student learning outcomes are achieved at reasonable costs. 

4.3 Literature Review 

A research report by Daniel Parent, “Labour Market Outcomes and Schooling in Canada: Has the 
Value of a High School Degree Changed over Time”, analysed the school-to-work transition 
process in Canada, focusing on the value of a high school diploma. Statistics Canada’s “School 
Leavers Survey and its Follow-Up” was used by Parent to derive details on labour market 
performance (earnings, jobs, and training histories etc.). The purpose of the study was to 
determine whether the value of a high school diploma (excluding the option value of pursuing 
post-secondary education) has markedly changed over the past 15 to 20 years. His conclusions 
were as follows: 

• the premium to holding just a high school diploma in Canada is substantially lower than in 
the United States; 

• labour earnings of high school graduates have stagnated and even decreased relative to 
those of dropouts, without major changes in the relative employment rates; 

• the earnings premium of a university degree has been increasing since the mid 1980s; 

• high school graduates’ labour market outcomes in Canada are essentially no better than 
those of high school drop outs, except perhaps in terms of employment rates; and 

• high school attrition was very sensitive to local labour markets. 

Robert Allen found, in his study “Education and Technological Revolutions: The Role of the 
Social Sciences and the Humanities in the Knowledge Based Economy” (November 1999), that 
there is a link between the level of education and employment success as reflected in 
unemployment rates. 

“Thus the highest unemployment rates are those of high school dropouts. They are followed by 
high school graduates and people with a technical or trade certificate and then by those with a 

college diploma The mediocre performance of those with a college diploma and the poor 

performance of those with a trade certificate is inconsistent with the basic view that specific skills 
guarantee a job in the new knowledge-based economy.” 
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Allen did note that those with some university education experienced considerably higher 
unemployment rates than those with a imiversity degree. The unemployment rate difference in 
1996 was 2.1% for women and 3% for men. 

He also reported that the percentage of Canadian workers in 1996 with managerial or professional 
occupations also depended upon academic credentials. For example, 71% of bachelor degrees 
graduates, 59% of those with some university experience, 48% of college graduates, 25% of 
those with trade certificates and 25% of those with only a high school graduation diploma were 
employed in a managerial or professional occupation. 

The data Allen presented in Tables 5 and 6 of his study indicate positive correlations between 
income levels and the level of education for both men and women. That is, incomes tend to rise 
from high school non-completes, to high school graduates, to trade certificates and college 
diplomas. 

Finally, in terms of total employment growth between 1991 and 1996 by level of education, Allen 
reported that total employment contracted for high school graduates and non-completes and trade 
certificates, while employment for those with a college diploma grew very strongly. 

John Mohammadi’s study, “Exploring Retention and Attrition in a Two-Year Public Community 
College” (VCAA Journal, Spring 1996, 39-50), attempted to explain the retention and attrition in 
two-year public community Colleges in the US in order to assess the usefulness of indicators of 
student retention and attrition for determining the effectiveness among community Colleges. 

Mohammadi noted that there are conflicting views of retention and attrition at US Colleges and 
that most studies do not take into account the external environment affecting student participation 
in College. 

According to Mohammadi, “There are two main reasons for this argument: (1) demographic and 
socio-economic factors relating to community college students are somewhat different from those 
relating to students attending four-year colleges. That is, on average, community college students 
are older, attend part-time more often, do not reside on campus, have lower degrees of goals, 
have lower high school grades, have more modest financial resources, are employed for more 
hours, have more family responsibilities, have relatively little interaction with other students 
outside of class, and are not strongly involved in campus activities and (2) external forces, 
particularly those related to community forces in the immediate geographical environment of the 
college’s service area, are also important in understanding and interpreting the retention and 
attrition patterns at two-year public community colleges. For example, the need for training and 
retraining of the work force for business, industry, and governmental agencies within any 
community college service area changes the social composition of the student population, thus 
requiring different definitions and methodologies to study student departure.” 

His principal conclusions were: 
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• the students’ goals for attending College are a very strong predictor of retention - 40% 
who left College after one year had no intention of completing a de gree or certificate 
program; 

• demographic and socio-economic factors, combined with an open access policy by the 
community Colleges, influence the attrition/retention rates - “Institutions that admit large 
numbers of less-well prepared students will tend to have low retention rates, regardless of 
how effective their retention programs are” - consequently, it is unfair to compare the 
retention rates of different types of institutions; and 

• college planners and government need to recognize that social forces in the immediate 
community directly influence retention and attrition. 

Mohammadi’s conclusions seem to speak strongly to the Ontario College retention issue, and 
were certainly supported by the feedback we received at our two Roundtable meetings with 
College Presidents and officials and in our other stakeholder interviews. 
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5.0 



THE FIVE COLLEGE CASE STUDIES 



5.1 Overview 

We undertook to examine the experience and the graduation rate statistics for five Colleges - 
Boreal, Canadore, Fanshawe, Humber and Sir Sanford Fleming. Officials at MTCU believed that 
some student tracking was going on with respect to program graduation rates. So one of the 
reasons the case studies were undertaken was to see how much individual student tracking was 
actually going on at the college level. 

Some of the key questions we posed to the college officials included the following: 

• What is your college’s capacity with respect to developing alternative methods of 
measuring attrition and graduation? 

• If your College had access to an individual student tracking system for calculating 
graduation rates, would it be helpful for strategic plaiming and programming? 

• How close is your College to actually tracking student graduation? 

• How much work would be involved in introducing a student tracking system? 

We requested examples of the data in the case where the College is tracking students by 
identification number. We also inquired as to the college’s interest in and capacity for developing 
a student tracking system for measuring attrition and graduation rates more accurately and 
whether the College had been tracking successful versus unsuccessful leavers. We inquired about 
the extra costs of developing this capacity. 

Here is how the graduation rate measures of these five Colleges compare to the total system and 
to each other. 





1997/98 Grad Rate 


Enrolment FTE Basis 


Boreal 


73% 


1,234 


Canadore 


61% 


2,658 


Fanshawe 


57% 


8,839 


Humber 


53% 


11,767 


Sir Sanford Fleming 


62% 


5,145 


Average: 5 Colleges 


61% 




Average: 25 Colleges 


55% 


168,779 


University Aggregate 


71% 


242,889 



Source: Data provided by Colleges Branch, MTCU 



We include program graduation rates for the five Colleges in the Appendix 3 section of the report. 
When we examine the graduation rate figures at a program level we discover that: 
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• The Colleges are tremendously diversified in terms of program offerings. Since separate 
programs have far different graduation rates, the comparisons of aggregate graduation 
rates between Colleges present some problems. 

• While many of the college programs are similar, there are also some distinct differences. 
For example, Humber lists 73 programs on its web site. Boreal 51 programs and Sir 
Stanford Fleming lists 60 programs. Clearly, the programs differ markedly, and of course 
this provides the job oriented college entrants with significant choice. 

The following table presents the distribution of full-time students by program at all Colleges in 



College 


Preparatory/ 

upgrading 


Visual 

and 

creative 

arts 


Human 

services 


Computing 

-electronics 


Business 


Health 


Tech. 


Algonquin 


4% 


12% 


13% 


21% 


29% 


7% 


14% 


Cambrian 


10 


10 


25 


12 


17 


9 


17 


Canadore 


12 


9 


15 


8 


31 


9 


17 


Centennial 


5 


3 


11 


18 


38 


8 


18 


Conestoga 


4 


6 


14 


12 


30 


12 


21 


Confederation 


9 


5 


21 


6 


27 


9 


23 


Durham 


5 


11 


17 


10 


33 


8 


16 


Fanshawe 


12 


13 


12 


8 


28 


7 


19 


George Brown 


3 


18 


14 


9 


28 


16 


13 


Georgian 


3 


12 


10 


8 


39 


12 


16 


Humber 


5 


17 


10 


14 


32 


11 


12 


Lambton 


17 


4 


19 


8 


24 


6 


22 


Loyalist 


6 


17 


26 


10 


20 


6 


16 


Mohawk 


4 


10 


13 


16 


24 


15 


18 


Niagara 


8 


12 


21 


13 


30 


6 


10 


Northern 


8 


2 


22 


11 


23 


11 


23 


St. Clair 


5 


7 


11 


11 


23 


15 


28 


St. Lawrence 


12 


3 


16 


16 


28 


18 


7 


Sault 


9 


4 


17 


14 


14 


12 


30 


Seneca 


4 


10 


11 


22 


38 


3 


13 


Sheridan 


7 


28 


16 


11 


28 


2 


7 


SSFleming 


2 


1 


25 


7 


22 


8 


36 


La Cite 


5 


16 


24 


20 


17 


7 


10 


Boreal 


9 


0 


22 


8 


20 


27 


14 


Grand Lacs 


0 


14 


17 


5 


45 


18 


0 



Note: Based on 1997-98 data (full-time students). 
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Ontario. Both Boreal and Canadore have 12% of their students in Preparatory/upgrading 
programs. Humber and Fleming, on the other hand, have only 5% and 2% respectively of then- 
students in these programs. Humber, Fleming, Canadore and Fanshawe have between 28% and 
32% of their students in Business programs, while Boreal has only 20% in these programs. Boreal 
however has 27% of their students in Health programs. None of the other four Colleges has more 
than 1 1% in these programs. Fleming has 36% of their students in Technology programs while the 
other Colleges have less than 19% of their students in these programs. 

In light of these differences, it is not surprising that graduation rates differ among these Colleges. 

5.2 Separate Comments Based On Discussions With College Officials 

Boreal 



Boreal is a relatively new Francophone College which began its operations in 1995. Boreal has 
more female than male students, and its ad min istrative system is unable to track the student 
population based on family incomes. However, the College knows that may of its students are 
presently receiving social assistance, and that the average age of the students is between 24 and 
29 years of age. Boreal’s students are heavily dependent on external financial aid. 

Since the families of the College’s students often have had little formal education, the College has 
had to introduce mechanisms to support student study skills at the entry level. The College 
emphasizes that a considerable number of its students are mature and in debt before they arrive at 
the College. 

The College has minimal student tracking in place, though a common student numbering system 
does exist that could be used to track individual students. The College does track students for 
graduation purposes. As in the other Colleges, Boreal’s students seem to be in motion; i.e. 
considerable program experimentation and movement. 

The College is proud of its common first year program in three areas: business, human services 
and technology. The College often has students finishing a two-year program with one diploma, 
then adding an additional year and receiving a second diploma. That is, after three years, they end 
up with two diplomas in two different programs. 

The College acknowledges the improved information flow associated with a student identification 
and tracking system. The College could not estimate the additional costs of implementing a 
student tracking system. And as in other Colleges, Boreal would like to see the individual tracking 
for graduation stretched out beyond six months of the normal program. 

A measurement problem might arise with respect to the common one-year program. In the above 
example, what is the actual point of entry for a student who finishes after three years with two 
different diplomas? This could be a serious problem in terms of measuring the graduation rate. 

Canadore 
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The officials pointed out that the Colleges are very different from the Universities. They 
emphasized three important differences: Colleges provide more program offerings than 
Universities; the student mix is very different in that Colleges have a higher proportion of older 
students; and finally, Colleges are not identical to each other as each College tends to focus on 
programs that they excel in. 

Canadore has a withdrawal form which it tries to have its “early leavers” complete. According to 
the information, students seem to withdraw primarily for personal reasons or financial need. The 
College also employs informal surveys of early leavers. But a large number of students leave 
without any explanation. 

It is interesting to note that Canadore has used a cohort measure of program graduation before it 
was introduced by the Ministry. Canadore found the measure useful because it assumed that any 
movement of students was roughly similar fi'om year to year. The College has the potential to 
create a historical cohort track record based on its own records, but this would be costly and 
require extra resources. 

It was recognized that the student tracking approach was superior to that of the cohort approach. 
Canadore could introduce a student tracking system based on its student identification number, 
but it would involve the development of some extra computer software and other costs. 

Finally, the College worries about the misuse or damage that could result from the dissemination 
of program graduation rates. On balance the College sees value to a direct student tracking 
approach for measuring graduation. 

Fanshawe 



Fanshawe, and the other Colleges, have a form that students fill out when they withdraw. But they 
do not have a survey of early student leavers. However, Fanshawe has a system for identifying 
students at risk based on the incoming grades of the students. This represents the full extent of 
student tracking. 

Fanshawe provided us with written answers to our structured questionnaire. Several excerpts are 
set out below. 

• 41% of Fanshawe student population enter College directly from high school, compared 
with the system average of 46%; 

• Many more students in College are married or sole-support; 

• Fanshawe does not use pre-admission testing for entrance to the College; 

• Many programs at Fanshawe now have common curricula in at least the first semester; 

• The College administered an exit survey in 98/99. In general, the results of the survey 
were inconclusive, with the majority of students stating personal and financial reasons for 
leaving; 
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• The anticipated extra costs (to shift to an individual student tracking system would include 
approximately one programmer for a year ($50,000) plus $10,000 for equipment. On- 
going costs would be approximately $10,000. 

Humber 



One of the College’s key officials has been researching the subject of retention and graduation for 
about 20 years. Consequently, the College has been tracking individual students and student 
leavers and has developed a software program that has been shared with other institutions. The 
College has been using a student tracking system for planning and programming. The College has 
also been measuring student aptitude at entry and at the point of graduation. 

On an empirical basis, Humber data suggest that approximately one-third of all first-year entrants 
leave the system. The one-third ratio fluctuates over time depending upon the strength of the job 
market. Clearly, in a strong cyclical upturn, the pull out of the Colleges into the job market is 
higher. Of those who leave programs and/or the College, roughly 40% of the leavers are regarded 
as successful leavers, with the other 60% as failures. 

Successful leavers are likely to be older students, with higher skill levels at entry, who were 
unclear about the college program delivering a job. . 

Sir Sanford Fleming 

This College has been involved in some extensive student tracking, and has calculated its own 
graduation rates by program based on the individual student number. The College is convinced 
that the individual student tracking method yields far better information. The College started the 
tracking program for entry classes in 1994-95, 1995-96 and 1996-97. Its tracking method 
identified conditional graduates and non-graduates, and students who graduate from different 
programs than they started in. 

The College recognized that this kind of information is superior and it was uncomfortable with the 
assumption that with the present cohort approach every thing evened out. The College also feels 
that the individual student tracking system to be imposed should be similar to the one used by the 
Universities. Finally, it is also important to understand and identify student mobility between 
programs and in and out of the College. 

The College does not have any follow up information with respect to the labour market success of 
its leavers beyond the KPI filed information. The College does, however, interview students with 
respect to reasons for withdrawal. The primary reasons seem to be personal, financial and/or 
program related. 

Sault Colle2e Of Applied Arts And Technology 

Since Sault College was unable to provide input though the Roundtable discussion, officials at the 
College wrote to Arthur Donner with respect to the College’s practices and interests. The 
information is included under this section. 
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The key issues identified were: 



• Students typically do not graduate in the time frame specified for their programs. The 
majority of three-year program registrants will graduate in a time frame beyond three 
years. 

• The number of programs that have future options available to the student has increased 
over the last five years. Students can opt for a different program after their first year. This 
causes difficulty in calculating a graduation rate based on the first year cohort. 

• Uncompleted courses at the end of a semester and subsequently finished, may cause the 
student to graduate after the peer group. Our system does not then include this graduate in 
the cohort. 

• GAS program is designed primarily as a feeder program and therefore should not be 
included in the Graduate calculation. 

• Transfers in and out of a program will change the cohort and therefore affect the 
graduation rate based on the present definitions. 

• If clustering is allowed on a College by College basis, there will be no system data that are 
comparable. 

• Our present client information system does not allow tracking by student. If a 
recommendation emerges using this type of tracking method, Sault College will not be 
able to comply. 

The College also put forth the following possible recommendations: 

• Have MTCU calculate a retention rate across the system by using the enrolment data 
already submitted for audit purposes and the graduate data also completed by audit. 

• Drop the graduation rate as a KPI. Concentrate on the employment rate and student 
satisfaction. 

• If retention is deemed as critical, concentrate on year-to-year retention as a measure of 
performance, not on the graduation rate. This would allow the system to generate data on 
why students leave and reco mm end solutions. 

• If the graduation rate is deemed critical, allow the time frame to exceed the program 
duration. 

• If a student tracking system is deemed appropriate, Sault College will require Ministry 
support to change its Information Systems to comply. 
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6.0 



CONCLUSIONS 



6.1 Measuring Student Success 

Most Colleges reject the status quo measure of graduation and attrition and accept that a new 
methodology is needed. 

There are three important issues regarding graduation rates - how should graduation rates be 
measured; what do comparisons of graduation rates across post-secondary institutions tell us 
about the labour market success of students who attend these institutions; and should graduation 
rates be used for funding allocation decisions by the provincial government? 

It is clear that using graduation rates as a measure of student success in comparisons across 
institutions may yield misleading conclusions unless one normalizes these rates for differences in 
the socio-economic demographic composition of the student bodies, mix and number of program 
offerings, labour market conditions and admissions standards. However, this normalization 
process would be difficult, if not impossible. In other words, one needs a multi-variate measure 
for student success. 

Furthermore, even if one could construct such a measure, what would be the appropriate 
comparator group of post-secondary institutions - only Colleges in Ontario, all Colleges across 
Canada, all Colleges in North America, or all post-secondary institutions in Ontario, Canada or 
North America? 

Cohort tracking systems may produce reliable measures for the graduation rates at the aggregate 
level if appropriate starting and end points are selected and a forward moving cohort 
measurement system is used. The current cohort systems used by Colleges in Ontario are 
inappropriate and produce graduation rates that are too low because they use too early a starting 
point for most programs, but more critically, too early an end point. 

If graduation rates, however they are measured, are used without normalizing for institutional 
differences, including their mandates and roles within the province, as a KPI for provincial 
allocation of funds to post-secondary institutions, they may distort planning at Colleges. In 
addition, this could also inhibit the ability of Colleges to adapt to changing economic and social 
conditions. 

Graduation rates, properly measured, can be useful for the Colleges for internal planning 
purposes. But without normalization and qualification, graduation rates that become public 
information may lead to incorrect decisions by students considering their post-secondary 
education and training options, in addition to inappropriate allocations, from an economic 
efficiency perspective, of government funding of post-secondary education and training 
institutions in the province. Colleges do not really know what appropriate graduation rates should 
be, so how can the public be expected to interpret these rates and make the right decisions. 
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Nevertheless, graduation rates may be turned to as a measure of success. So it is imperative that 
the measurements be improved. It is also important that they not be used without any 
qualifications or adjustments. 

One could attempt to normalize graduation rates through the use of econometric techniques. That 
is, one could use a cross-section and time series data set consisting of graduation rates (measured 
in the same way for all post-secondary institutions in the province); demographic, education and 
employment characteristics of the student bodies; local economic conditions; and one or more sets 
of dummy variables for each post-secondary institution in the province. The dummy variables 
would reflect differences in the roles of these institutions. Regressing the graduation rates against 
these other variables would provide a means for normalizing graduation rates for each institution. 

While the resulting normalized graduation rates would be useful as a KPI for budget allocation 
decisions by the province, they still may be weakly correlated with student or institutional success. 
Hence, some other measures should be developed that together with normalized graduation rates 
provide more comprehensive indicators of student and institutional success. 

6.2 Graduation Rates as an Indicator of Success for Colleges 

Colleges are more unique than Universities. Students entering the Colleges from high schools are 
more immediately required to make career choices than their counterparts entering Universities. 
That is, to a greater degree than is possible at the college level, students entering University can 
select a number of courses in their first year before they decide to specialize. 

Moreover, the graduation rate is only one output measure, and downplays the measures of inputs. 
Open and easy access to Colleges compared with Universities makes a big difference on the input 
side. Open and easy access is regarded as a societal goal. Colleges do not require high school 
OACs. As well, the graduation rates of Colleges may be biased downwards because they accept a 
relatively high proportion of special needs students who are less likely to graduate. 

Success at the college level involves more than graduating students. In some programs, 
graduation and a certificate are necessary to qualify for a job. For other jobs, formal credentials 
are not required. Only some degree of training and work experience are necessary. 

The graduation rates as currently measured by the Colleges are most likely imderestimated in 
comparison to the graduation rates produced by the methodology used by the Universities in 
Ontario. 

Comparisons of the aggregate graduation rates between Universities and Colleges at present 
would be misleading for a number reasons. Comparisons of program graduation rates at Colleges 
to graduation rates at program levels in Universities would be completely misleading. The 
introduction of a student tracking system at the college level would help provide for more 
meaningful comparisons. 
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Indeed, mounting a pilot program involving a small number of Colleges would provide useful 
information to determine whether the introduction of a student tracking system similar to the 
Universities would be useful and cost effective. 

However, whatever system is used to measure graduation rates for the Colleges, it is important to 
use appropriate entry points (the time when a student actually begins a program) and end-points 
(the elapsed time for a student to have completed a program). This also suggests that there would 
be value in a pilot program where each College experimented with different entry and graduation 
time frames. 

It is important to keep in mind that success is multi-faceted, so there is a need for other measures 
besides successful graduation. Some of these measures may have to be institution-specific since 
Colleges differ in many ways. In developing other indicators and a better measure of graduation 
rates, historical context will be very important. A one-size fits all approach may not be 
appropriate. 

Graduation rates should be expected to differ across Colleges and programs and between 
Colleges and Universities. However, there probably is some kind of steady state or base case 
level of measured differences. If one could normalise for these differences, then one could 
start to determine how to improve upon the differences. 

Current aggregate college graduation rate measures can mislead because of differences in 
demographics, different admission standards, different mix of programs and courses, and different 
labour and economic market conditions. The public rarely considers all of these variables in terms 
of reflecting upon the meaning of an aggregate college graduation rate figure. 

What all of this suggests is that even comparisons of aggregate graduation rates across different 
Colleges is somewhat misleading without a long track record of graduation rates to compare. 
Moreover, the same case can be made for University comparisons within their system. 
Comparisons across Colleges and Universities, while perhaps interesting, provide very little useful 
public policy insight. 

6.3 KPIs 

The numeric approach associated with KPIs is intended to provide a useful tool for students 
selecting between different Colleges and between Colleges and Universities. At the same time, the 
numerical approach allows the MTCU to evaluate whether each College is achieving its objectives 
as reflected in large part in graduate employment, graduate satisfaction, student satisfaction and 
employer satisfaction. 

Implicit is the thinking that somehow the KPIs can capture best practice delivery of programs, 
training, and ultimately jobs. But in broad brush terms, the numerical approach associated with 
KPIs suggests a level of precision that may not really be appropriate. 
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For example, when linking KPIs to the labour market opportunities of graduates (the most 
important final outcome), it is clear that there is a significant time frame problem. That is, the 
effects on labour market outcomes have to be measured at an individual level and over time. But 
there are many “non-controllable variables” which are also determinants of labour market success 
- ethnicity, socio-economic background, region, pre-school family incomes, education levels of 
parents, literacy, etc. 

As well, there is the issue of the quality of the working career post graduation from College; for 
example, income levels over time, the stability of the job, the nature and flexibility of the job (part- 
time, full time, self employment, etc.). 

But there is a strong case to be made that the college graduation rate may not truly reflect labour 
market success. On the positive side, it might well be that well structured KPIs are a useful 
catalyst for change with respect to all of the stakeholders and could lead to more focused action 
with programs moving in the right direction. 

Each college recognizes its obligation to its many stakeholders to provide assurance that its 
activities are inclusive of and consistent with its specific values, goals and strategic direction. 
External accountability processes could provide objective means by which those who use, support 
and fund the Colleges can be assured that their confidence is well placed. 

6.4 Recommendations 

Measurement Of Graduation Rates 



Graduation rates can be estimated for individual programs, for clusters of programs and/or at the 
aggregate institution level. Graduation rates for individual programs or clusters of programs will 
be useful for internal planning by each College. Institutions need to monitor what happens 
internally from semester-to-semester and among programs. 

But graduation rates for individual programs can be misleading for students making decisions 
regarding institutions and programs and may be inappropriate for making funding allocation 
decisions across post-secondary institutions. The links between student/institution success and 
graduation rates are even more tenuous at the program level than at the level of the institution. 
Consequently, if graduation rates are to be used as one of the KPIs for funding allocation 
decisions by the province, college-wide graduation rates are preferred relative to the program 
graduation rates. 

In addition, these rates should be normalized using the econometric methodology we outlined 
above. 

The graduation rates can be calculated using any one of the following three models: 

• cohort tracking: group A at time X and compare to group B at time X+Y ; 

• cohort tracking: group A at time X and compare to group A at time X+Y ; 
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• individual student tracking: each individual starting at time X and compare to same set of 
individuals at time X+Y. 

In the absence of a pilot program comparing cohort model 2 with individual tracking (model 3), 
we cannot choose between these two models. But either one is preferable to the current approach 
- cohort model 1 . 

Over time, we would suggest moving to a system where students are given a provincial student 
number so that they can be tracked if they move from one institution to another. In other words, 
individual student tracking across the entire post-secondary education/training system in the 
province is the preferred model in the long run. 

If graduation rates are to be estimated using individual student tracking, the province 
should subsidize, through a one-time cash payment, the costs for each College to acquire 
and develop the needed computer hardware and software. Only the province, using its 
financial levers, can ensure that a uniform system - computer software, similar length 
student identifiers - will be developed by all the Colleges. 

A pilot study involving a small number of Colleges should provide a reasonable estimate of 
the upgrade costs for all Colleges. Of course, the financial assistance by the province should 
involve new money and not arise from a reallocation of existing operating and capital 
grants. 

At this time, it appears that the following Colleges would have some difficulty in moving quickly 
and at minim um cost to student tracking system based on institutional individual student numbers 
for calculating graduation rates - Boreal, Confederation, Durham, Fanshawe, Lambton, Northern, 
St. Clair and Sault. 

As for the time periods - X and Y - we recommend that X begins after the first semester 
following the entry of a student into a program. This will exclude those students who are 
uncertain about whether or not they made the right decision to go to College and about whether 
they selected the right program. Further, we recommend that Y be at least 1 50 percent of the time 
period for completing a program (this happens to be the official US NCES norm) - 4.5 years for a 
three-year program; three years for a two-year program and 1 .5 years for a one-year program. 
Furthermore, feeder programs and remedial programs should be excluded from the graduation 
rate calculations. 

The combination of a pilot program and Y years of tracking the first group of students means that 
it will take at least five years before any meaningful information will flow from the new 
measurement system. The pilot project however, will provide in a much shorter time frame an 
indication of how reliable the existing cohort tracking method is in generating estimates of 
aggregate, institution level graduation rates. Assuming a forward moving cohort system produces 
reasonable estimates, then it may be appropriate for the government to have greater confidence in 
the normalised graduation rate figures for planning purposes. 
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Nonetheless, it is crucial to remember that even the current five KPIs together may not provide an 
accurate measure of the success of each College in achieving its goals. Other KPIs, some of them 
along the lines of those used in BC, may be needed in order for funding allocations to create the 
right incentives for Colleges. 

Pilot Projects 

The provincial government should sponsor and underwrite the costs of several pilot studies. A 
small number of Colleges (3-4) should be selected to develop individual student tracking systems. 
The Colleges should experiment with different entry points and time frames for measuring 
graduation rates. In order to provide information more quickly, the Colleges should apply the 
system to historical data; that is, they should go back four or five years if possible. 

The Colleges should differ in terms of size (student body, number of programs), current level of 
sophistication in tracking students and geographic location. The roles of the pilot projects should 
be as follows: 

• to see if the additional costs of a more complex tracking system are warranted by the 
production of more reliable estimates of graduation rates; 

• to estimate the system-wide costs of moving to a more complex individual student 
tracking system; 

• to examine the impacts on measured graduation rates of different assumptions for starting 
and end points; and 

• to see whether there is a high correlation between graduation and labour market success. 

The MTCU should not msh out to create a new system that might be too costly and/or produce 
marginally better estimates of graduation rates and other measures of student success. There is a 
need to experiment to find the most appropriate measurement methodology. The MTCU should 
also examine the feasibility and costs of moving towards provincial student numbers and province- 
wide tracking of students. 

6.5 SUMMING UP 

The graduation rates and the attrition rates are important public policy educational indictors. But 
at best they represent only rough proxies for student success in the labour market. 

Attrition rates are important financial indicators for the Colleges. To some, attrition symbolises 
the wastage of educational investment funds; whereas in fact, some attrition may be a healthy sign 
of a strong job market. 

Consequently, there is a need to improve the data on graduation/attrition rate measures at the 
college level. Experience suggests that the optimal system would track students directly. 

The Ontario government allocates about $700 million annually to the Colleges, and thus the 
attrition information is useful, and relative to the total budget, the extra funds for improving the 
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measuring of graduation rates and for developing other measures of success should be found. 
There is a strong case for investing more money in the measurement of graduation rates so as to 
improve on the quality of the data on a program basis. 

Note that the government surveys college graduates for their employment records. There is no 
comparable survey of attrition (successful or unsuccessful in job market terms) from the system. 
There is a need for good data relating to why students leave and their subsequent employment 
records. 

In closing, 

• While the four KPI measures that are tied to funding at the college level are useful to all of 
the stakeholders, the current measure of graduation at the college program level is flawed 
and misleading. 

• The aggregate measure of graduation at the college level is probably reasonably 
representative of the total picture. However, comparisons across Colleges can be 
misleading, particularly since there is no attempt to normalize the data for differences in 
programs, location, and/or the demographics of the student population. . 

• The introduction of a university style measure of graduation at the program level (i.e. 
following individual students rather than cohorts of students) would be an improvement 
over the current system. But recognize that using a college institution identifier will still 
have the problem of capturing and measuring students moving between different Colleges 
and in and out of the college system programs (either to the job market or to other post- 
secondary educational opportunities). 

• ACAATO should propose that MTCU move towards introducing a student tracking 
measurement system for graduation based on the Colleges’ student identification numbers. 

• To ensure that the graduation rate data at the program level are reliable and not 
misleading, ACAATO should suggest that MTCU sponsor several pilot studies, 
experimenting both with measurement and interpretation of the data. While the university 
measurement system based on student tracking is superior to the current college cohort 
basis measure, it is also clear that even the university style system has some problems. 
Consequently, The Ministry is advised not to rush out to create a new system that could 
also mislead. There is a need to experiment to find the most appropriate measurement 
approach. 

• ACAATO should also petition MTCU to undertake a survey of early leavers from college 
programs, attempting to identify successful from unsuccessful college leavers in terms of 
the job market. 

• How long will it be before meaningful graduation information flows from a new system - 
at least five years after the new system is introduced. 
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• Finally, the best system would be based on provincial student identification numbers, since 
this would allow the Ministry to track students in and out of of post secondary institutions 
and in and out of the labour market. 
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